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(54) Document processing 

(57) A tag-based user interface scheme for digitiz- 
ing and processing hardcopy documents utilizes a stick- 
er (1 1 0) that includes a printed data code representative 
of a user identity code (118) and a senrice code (112). 



When the sticker is applied to a hardcopy document and 
scanned, the sticker is kxated, the data code is parsed, 
and a desired senrice Is perfomied based upon the in- 
fonnation stored in the data code. 



< 

in 



Q. 
Uj 



The Person- 




PaperWare 



FIG. 1 



Printed by Jouw. 75001 PARIS (FR) 



EP 1 001 605 A2 



Description 

[0001] The invention relates to a hardcopy-based computer user interface schenne for document processing appli- 
cations. 

[0002] Many people are most comfortable dealing with documents in hardcopy format. In general, hardcopy docu- 
ments are easier to read, handle, and store than documents kept in the digital domain. No special expertise or computer 
hardware is needed. 

[0003] However. In general, it Is far easier to manipulate documents in the digital domain. Text can be indexed, 
searched upon, reformatted, extracted, and othenwise changed. Stored documents can be easily duplicated, without 
loss of quality, and transmitted from person to person (lor example, via e-mail). And significantly, all of this can be 
accomplished without using paper. Moreover, digital copiers and scanners are becoming far more prevalent in both 
office and home settings. 

[0004] On the other hand, handling documents in the digital donnain typically requires access to a computer system 
or network. If the user of the computer system does not have a baseline level of expertise or competence In using the 
system, then productivity can suffer. This is a serious impediment to the Implementatton of a "paperless office, ' in whteh 
digital documents are the prevailing document type 

[OOOS] Accordingly, there is a need to be able to effectively manage documents in the digital domain, as well as to 
ease the transition from hardcopy documents to digital documents. 

[0006] Previous attempts to facilitate handling digital documents have often used traditional user-interface para- 
digms. For example, when a hardcopy document is to be scanned and entered into a document repository, commands 
to that effect are first entered into a connputer terminal or scanning device, which then performs the desired sen/ice 
with the document. A similar sequence of steps is performed when the hardcopy is to be scanned and faxed, scanned 
and e-mailed, scanned and recognized (via optfcal character recognitwn software), or any of numerous other possi- 
bilities. Although the entry of commands can be facilitated via user-friendly software or self-explanatory commands, 
these extra steps are still tedious and may still require a certain level of expertise. Moreover, the sequence of commands 
entered may be fc)st once the operatkjn has been performed, and there is a potential for en-or even with experienced 
users. 

[0007] Another possibility is to employ a cover sheet that includes a form tor specifying commands. The cover sheet 
Is filled out as the user desires (either by hand-writing commands or by marking check-boxes, for example), and the 
scanner Interprets the commands on the cover sheet and processes the following document accordingly. This, too, 
can be tedraus and'relatively Inefficient, as It requires a special-purpose cover sheet to be used for each job. It can be 
Inconvenient to maintain a supply of the proper cover sheets. 

[0008] Various one- and two<Jimensional data codes are known and available to be used to store digital data on 
hardcopy documents. For example, various types of barcodes (for example, the familiar UPC symbol used as a retail 
product code) are very well known and are robustly decodable. Other examples of linear barcodes are known as Code 
39, Code 1 28, Interleaved 2 of 5, and Postnet. Tw<Klimensional codes, such as the PDF417 code and the UPS Max- 
iCode used by the United Parcel Service to track packages, ifor example, are becoming more and more widespread, 
[0009] Self-clocking glyph codes, such as Xerox DataGlyphs, are attractive for embedding machine-readable digital 
information in images of various types, including ordinary hardcopy documents. These codes have substantial tolerance 
to Image distortion and noise because the digital infomiation they encode is embedded in and fully defined by explicit 
machine-readable marks (I.e.. 'glyphs," a term used herein which Is not intended to be limited to Xerox DataGlyphs, 
but rather is intended to cover all machine-readable marks). These glyphs not only encode the Informatkan that is 
embedded in the code, but also define the sample clock that Is emptoyed to extract that Informatton from the code, so 
they are responsible for the "self-ctocking" property of the code as well as thedistortfon and noise tolerance. 
[001 0] Another known advantage of self-clocking glyph codes is that they ordinarily have an unobtrusive visual ap- 
pearance. This is especially so of codes composed of glyphs that are written on a two-dimensional spatially periodic 
pattern of centers, such as a regular lattice-like pattern of centers, because the spatial periodicity of the glyphs causes 
the code to have a more-or-less unifomily textured appearance. For example, logically ordered single bit digital quanta 
typically are encoded by respective elongated slash-like glyphs which are written on a two-dimensional, spatially pe- 
riodic pattern of centers in accordance with a predetermined spatial formatting rule, with the individual glyphs being 
tilted to the left or right of vertical by approximately +45* and -45" for encoding togical "O's" and "1 's", respectively. The 
mutual orthogonality of the glyph encodings for the two k)gk:al states of these single bit digital quanta enhances the 
discriminability of the code sufficiently to enable the embedded information to be recovered, even when the code pattern 
is written on a sufficiently fine grain pattern of center to cause the code pattern to have a generally uniform grayscale 
appearance. It, however, is to be noted that it also has been shown that self-clocking glyph codes can be designed to 
encode multi-bit digital quanta in the glyphs. 

[0011] Accordingly, it wouW be desirable to provide a solutkjn that facilitates the use of documents in the digital 
domain and the transitkjn of documents from hardcopy to digftal formats. Such a solutton should be simple, efficient. 



EP 1 001 605 A2 



convenient, and require little or no expertise on the part of the user. 

[001 2] In accordance with the present invention, a method for processing a document based on information in a user 
Interface tag comprises: 

scanning the document to produce an image representative of the document; 
locating the user interface tag in the image; 
decoding data represented in the user interface tag; 
associating the data with a service; and 
perfonning the specified service. 

[001 3] Thus, the Invention provides a tag-based user interface scheme adapted to perform sen^ices with a hardcopy 
document dependent on user or service Informatbn stored as printed data on a tag associated with the document. 
[0014] The present invention uses a scheme of encoded tags, such as adhesive stickers or labels, to serve as the 
primary user interface in a hardcopy document processing system. Such a system would pemiit a user to specify an 
action or a sewice to be performed and his own identity simply by applying a sticker to the document and placing It 
into a bin to be scanned. 

[001 5] The stickers are relatively smalland unobtrusive, and In an embodiment of the invention use Xerox DataGlyphs 
to encode the user's identity, a desired serotee, and an optranal argument for the sen/ice. The user maintains a supply 
of stickers corresponding to a particular sewlce (e.g.. "scan and send to my personal e-mail account"). When the user 
desires that service to be performed, he simply applies one of the appropriate stfckers to the document. 
[0016] An example of a computer system performing the inventbn operates by first accepting Information on the 
user's kJentity. This can be obtained, for example, by scanning and analyzing one of the user's business cards. This 
information Is stored In a database and is given a unique user ID number. The user ID is combined with a desired 
service code, and the resulting data set is encoded Into the desired printed data format. The system then prints a 
customized set of labels representing the user ID and servne. 

[0017] When the user then wishes to have the servkje specified by his stickers performed, he simply applies one of 
the stickers to a document and places it into a scanning queue. The document is scanned, the sticker is Identified and 
decoded, re-associated with the user's informatkxi retrieved from the database, and the desired service is performed. 
[0018] As will be recognized, this system can be adapted to perform numerous services and actions, including but 
not limited to simply scanning and storing the document in a repositoty, faxing the document, converting the document 
into a standard electronic format (such as Microsoft Word format), and others. 

[0019] Accordingly, the sticker-based user Interface of the present Invention Is convenient and simple, as it does not 
require the user to enter commands on a computer system or fill out a cover sheet for every document to be processed. 
It is efflcient, in that it is only necessary for the user to keep a supply of small stickers. 

[0020] Some examples of methods and stickers according to the inventkxi will now be described with reference to 
the accompanying drawings, In whlch:- 

Flgure 1 Is an exemplary glyph sticker; . 

Figure 2 Is an exemplary block diagram illustrating a data stnjcture represented by the glyph sticker of Figure 1 ; 
Figure 3 Is a system-level block diagram; 

Figure 4 is a high-level functional flow chart of the sticker creatksn function of the invention; 

Figure 5 is a high-level functional flow chart of the sticker processing function of the invention; 

Figure 6 is a flow chart illustrating the sequence of steps perfonned in locating the stfcker as set forth in Figure 5; 

Figure 7 Is a flow chart illustrating the sequence of steps performed in decoding the sttoker as set forth in Figure 5; 

Figure 8 Illustrates a portion of an exemplary glyph lattico with associated glyph bitmaps and lattice vectors; 

Figure 9 Is a flow chart illustrating the sequence of steps performed In determining the glyph lattice as shown in 

Figure 7; 

Figure 10 is a schematic diagram illustrating the search directkjns employed In searching for glyphs as shown in 
Figure 7; 

Figure 11 is a flow chart illustrating the sequence of steps performed in finding a seed glyph as in Figure 7; 
Figure 12 is a schematic diagram illustrating the procedure used to find all glyphs as in Figure 7; 
Figure 1 3 is a flow chart illustrating the sequence of steps perfonned in finding all glyphs near the seed glyph as 
shown in Figure 7; and, 

FIGURE 1 4 is a flow chart illustrating the steps perfonned in decoding a glyph pattern. 

. [0021] Figure 1 illustrates an exemplary glyph sticker pattern 110, which serves a paper user Interface tag according 
to the invention. It should be noted that this pattern 110 is not to any particular scale; it is contemplated to be on the 
order of 1 inch (2.5 cm) or less in the vertical direction; it has been substantially enlarged here for clarity and to facilitate 
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the identification of details. 

[0022] Furthermore, while the user interface tag of the invention is illustrated in Figure 1 as an adhesive sticker 
capable of being applied to documents, it should be observed that the pattern 110 may also be applied to forms, slip 
sheets, cover sheets, business cards, identification cards, name tags, and other objects capable of being associated 
with a document and further capable of bearing machine-readable information. For example, in an alternative embod- 
iment of the Inventbn, lapel pins bearing the pattern 110 may be associated with a document by scanning the lapel 
pin (e.g., with a hand-held laser scanner) at the same time the document is being scanned and processed. 
[0023] As shown, the glyph sticker pattern 110 Includes several important attributes. A substantially rectangular 
border 1 1 1 surrounds the remaining features, and is In turn surrounded by white space 1 1 3. A glyph field 1 1 2 contains 
a printed representation of digital data used to perform the goals of the inventkxi; the stnjcture and contents of the 
glyph field 112 will be discussed in further detail below. For now, it should be noted that the glyph field 112 of Figure 1 
is shown as using Xerox DataGlyphs. However, it will be obsen^ed that any recoverable printed representatksn of digital 
information, including but not limited to bar codes, one- and two-dimensional data patterns, and optically recognizable 
alphanumeric characters can also be used in alternative embodiments of the invention to similar effect. 
[0024] Other kJentifying elements may also be present within the glyph sticker pattern 1 1 0. For example, the Xerox 
"Digital X" 114 and the registered mark "Papenrare®" 116 are present and are used for aesthetic purposes and as 
tradenrarks. The phrase "The Person' 118 Is also present; it is contemplated that this field nrtay be replaced with an 
indication of the user's name or other infomialton for visual Identification. There Is also a border portion 120 that Is 
representative of a folded page. This border portion 120 nf«y be present for aesthetk: purposes only, or may be used 
by the inventbn to facilitate determination of the conect orientation of the glyph sticker pattern 110. This aspect of the 
inventk)n will be discussed in further detail bekiw. 

[0025] Referring now to Figure 2, an exemplary data structure 210 embodied by the glyph field 112 is set forth as 
shown. The data stmcture 210 includes a sewlce code 212. whfch in one embodiment of the Invention is a single byte 
(eight binary bits). Accordingly, the sen/tee code 21 2 can represent up to 256 different possible acttons, transformatkjns, 
and sewfces. Exemplary services may Include, but would not be limited to, "scan to document repository," "scan and 
send via e-mail." "scan and fax," "scan and print copies," and so forth. An indicated sewfce may, without limitation, 
include a plurality of actions (e.g., scan, then recognize characters, then e-mail the text), and may also involve trans- 
fonnatran of the document from hardcopy to electronic fomi, and possibly back to hardcopy form. Moreover, one or 
more possible actions might involve the removal or alteration of the glyph sticker pattern 11 0 on the digitized image of 
the document, as its presence is no longer necessary (and may detract from the document's appearance) once it has 
been acted upon by the system and method of the Invention. 

[0026] A sen^ice argument 21 4 provides a coded argument to the service code 21 2. In an embodiment of the inven- 
tion, the argument 214 Is an additional byte (eight bits) of information. For example, certain sewtees may require a 
numerc argument (e.g.. "scan and print copies" followed by an argument of "lO" will print ten copies), aher sen/tees 
may require a differently coded argument (e.g., "Scan and fax" foltowed by an argument of "2" may represent a command 
to fax the document to the user's home fax nuntber, as opposed to an office fax number or, perhaps, an alternate office 
fax number, both of whch would have different argument numbers). Many other uses of the servk;e argument 21 4 may 
also be envisioned without departing from the scope of the invention. 

[0027] An identity code 216 comprises most of the rdst of the data structure 21 0. In an embodiment of the invention, 
the identity code includes sixteen bytes of identity informatbn (sufficient to encode a unique identification number for 
each member of nearly any conceivable populatron) and eight bytes of security infomnation, the latter rendering the 
klentlty code essentially tamper-proof. The Identity information is fomned from the network address (either an Ethernet 
address or an IP address) of the machine Issuing the identity code, a time stamp, and a sequence number Although 
a number formed in this manner is not guaranteed to be unique under all circumstances (for example, if the database 
of sequence numbers is lost and the same time stamp is coincWentally used twrice), it is quite robust and sufficiently 
unique for the purposes of the inventron. It shoukJ also be noted that the kJentity code may be formed in any of numerous 
other ways, all of which would be acceptable for use with the present inventkxi. 

[0028] The data structure 21 0 also includes error-correction and synchronization infonnation throughout the specified 
data fiekJs in a manner well known in the art and not shown in Figure 2. 

[0029] Figure 3 represents a system-level oven^iew of the inventran, broken down Into several high-level functional 
elements. A "front end" portton 310 receives infomriatlon from a data input 312. The data Input 312 may comprise a 
keyboard, a business card scanner, a votoe recognition device, or any other means of data collectton operative to 
receive data representative of the user's ktentity. The data input 312 may be expected to receive the user's name, 
company, title, address, one or several telephone numbers, one or several e-mail addresses, as well as other possible 
identification information (such as a U.S. Social Security number or a conference ID number). This user information is 
then received by an identity processor 314, which stores the infomnatton as a record in a database hosted by a database 
server 31 6. Concurrently, the user intomiation is processed by the kJentity processor 31 4 to produce the identity code 
216 (incorporating the kJentity and security information discussed above). The identity code 216 and any desired op- 
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tional service code 212 and service argument 214 are then encoded and printed on a printer 318 or other hardcopy 
output device in communication with the identity processor 31 4. 

[0030] When the user wishes to process a document, he attaches a glyph sticker according the invention onto the 
document and places the document into a scanner 320. which Is part of a "back end" 31 9 of the system. The scanner 
B Is preferably an automated scanner capable of handling multiple jobs and multi-page documents without user inter- 
vention. However, any other type of digitizing apparatus (such as flatbed scanners, digital copiers, and hand-held 
scanners) wouW also be usable In this application. The scanner 320 reads the document and formulates a bltrriap 
representative of the document (and the glyph sticker on the document). 

[0031] An action processor 322 reads the bitmap received from the scanner 320. Identifies and decodes the glyph 
10 sticker, and accesses the database sewer 316 to deteimine the identity of the user. The desired service may be known 
from accessing a list of possible services stored kwally on the actton processor 322. or may also be determined by 
accessing a serofce database on the database sewer 316, or altematlvely nreiy be Inferred simply from the identity of 
the user. 

[0032] Based on the user's identity and the desired sewioe. the action processor 322 then causes the desired action 
IS to be performed, which may involve the generation of a transformed document by an output device 324. The output 
device 324 is characterized generally here, but as discussed above, may comprise a hardcopy printer, a facsimile 
machine (or modem capable of sending fax messages), a network connection for e-mail, a connectton to a document 
repository, a digital storage device (such as a floppy disk drive), or an aggregation of some or all of these and other 
functtons. 

20 [0033] While the system of Figure 3 is illustrated as being broken down into several lunctkjnal blocks, it shouW be 
noted that the functions depicted can be performed on a single computer system or by several computer systems 
connected via a communications network. As a specific example, the database sewer 31 6 may be local to the identity 
processor 31 4 and the action processor 322, or it may be a separate processor in communication with the other ele- 
ments via a communications network, such as the Internet or a corporate intranet. 

[0034] As suggested above by Figure 3. the front end processing of the system, namely the creation of glyph sticker 
patterns 110 based on user nfomration and a desired servfce, takes place as a sequence of steps. These steps are 
shown in the flow chart of Figure 4. 

[0035] First, the user (or another person) inputs infomiation (step 410) into the system, typically via the data Input 
312 (Figure 3). All of the information is then stored in the database (step 412) on the database server 316. A sewice 
30 is then selected (step 41 4). The service may be chosen from a "menu" of possible services, or in one embodiment of 
the invention, only a single service may be operative. In the latter case, the service code 21 2 and the sewice argument 
214 (Figure 2) may optionally be omitted from the glyph sticker pattern 110. 

[0036] The user Identity code 21 6, the sewlce code 21 2. and the sewice argument 21 4 are then encoded and formed 
into a glyph field 112 representative of the hformatkxi (step 416). The newly-created customized glyph fieM 112 is 
35 then printed any number of times as glyph sticker, pattems 110 onto any desired number of stckers (step 418) for 

eventual use by the user. 

[0037] Figure 5 is a flow chart illustrating what happens after the user has applied a stfcker to a document and 
presented it tor scanning. First, the document Is scanned (step 510) and the resultng bitmap Is retained for processing. 
The original hardcopy document nray. at this time, be returned to the user, as it is no kxiger needed by the system of 
the invention. 

[0038] The glyph stfcker pattern 110 Is then kxated on the document (step 512). In a preferred embodiment of the 
Inventkjn. the glyph sticker pattem Is located via a rectangle-matching method described below in conjunction with 
Figure 6. In an alternative embodiment of the invention, the user is directed to place the glyph sticker at a certain 
locatkjn on the document (e.g. , the upper right comer of the first page), thereby simplifying the operations required to 

« locate the glyph stfcker pattern 110. In another alternative embodiment, the sticker pattern 110 can be identified any- 
where on a document by segmenting the scanned document bitmap into text and graphics portions, and in the line- 
graphics portions, locating the right-angled lines that comprise the border of the glyph sticker pattern 110. A method 
for accomplishing this is set forth in detail in U.S. Patent No. 5,202.933 to Bloomberg, entitled "SEGMENTATION OF 
TEXT AND GRAPHICS," which Is hereby incorporated by reference as though set forth in full. To facilitate this, half- 

» tones and finely textured graphics regions (such as lithographed photographs) may also be segmented out (and away 
from the line graphics) via the methods taught In U.S. Patent Nos. S.065,437 and 5.131.049. It should be noted that 
the glyph sticker pattern 110 may be present in nearly any orientation; accordingly, the segmentatton method should 
be adapted to check several different orientations (by using differently-shaped structuring elements in the morphotogical 
operations used in the incorporated "933 patent document to identify horizontal and vertical lines) before it isdetemiined 

ss that no sticker is present. Finally, a Hausdorff-dlstance-based algorithm may also be used; it should be noted that this 
method, like the morphological method referenced above, is rotattonally sensitive and hence relatively computattonally 
inefficient. 

[0039] Once the glyph stfcker pattem 110 has been kxated. the data within the glyph field 11 2 is then decoded (step 
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514). This is accomplished via methods that will be discussed in further detail below, in connection with Figures 7-14. 
The data is then associated (step 51 6) with user identification intonmalion (received from the database server 31 6) and 
a desired service, and the service is perfomned (step 518) be means known in the art and discussed above. 
[0040] A presently preferred method used to locate user interface tags (e.g., the glyph sticker pattern 11 0 of Figure 

s 1 ) within a digitized inrage is illustrated in Figure 6. This method .can be used to identify and locate objects that are 
approximately rectangular in shape, such as the iconic representation of a page with one comer folded over, as used 
as the glyph sticker pattern 110 In an embodiment of the Inventton (see Figure 1). Such rectangular objects can be 
characterized by a height dimension and a width dimension. The Idenllflcatlon method can Identify the desired pattern 
in any orientation and with relatively small scale variations (such as those occurring In the printing and digitizing proc- 

10 esses), even when image noise is present. 

[0041] The method is operative on monochromatic images (i.e., binary images represented with only one bit per 
pixel) . If the digitized Image is In some other fomfiat (such as color or grayscale), it should first, as a precursor to the 
method set forth in Figure 6. be converted to a binary format, typically via a threshold function or by dithering. 
[0042] The characteristics (e.g., resolutfon) of the digitizing device used are expected to be known Therefore, be- 

'5 cause the absolute size of the glyph sticker pattern 1 1 0 (Figure 1 ) is known, its image size can be cateulated. Accord- 
ingly, the expected width wand the expected height /j of the pattern 110. in pixels, is either known or can be easily 
calculated. With width and height tolerances (Aw and A/j, respectively) to account for minor variations In scale, as 
stated above, a minimum expected width would be = {w- Am), a maximum expected wkJth woukl be = (w 
+ Avv), a minimum expected height would be h„,„ = (h - Ah), and a maximum expected height wouW be =(h + 

20 A/)) . In a presently preferred embodiment, A/i = (0.1)/jand &.w=(OA)w, stated another way, the tolerances are 10% 
of the respective height and wkJth. It shouM be noted, however, that other tolerances ranges may also be operative, 
and that different tolerances may be used for the width and height dimensions. 

[0043] Accordingly, and based on the foregoing, the minimumand maximum expected diagonal measurements would 



"min = -/(w- Aw)^ + (/)-A/))^ and, d^^ = A/(w+A^v)^ + (h + A/))^ 

respectively. 

[0044] After a suitable digitized image is available, all connected components within the image are initially identified 
(step 610). A connected component is a set of pixels of a single value (e.g.. the value representing black), wherein it 
is possible to form a path from any pixel of the set to any other pixel in the set without leaving the set (e.g. . by traversing 
only black pixels). In general terms, a connected component may be erther "4-connected" or "8-connected." In the 
4-connected case, the path can move in only horizontal or vertkal directtons, so there are four possible directions. 
Accordingly, two diagonally adjacent black pixels are not 4-oonnected, unless there is another black pixel horizontally 
or vertically adjacent, serving as a brWge between the two. In the 8-connected case, the path between pixels may also 
proceed diagonally. A presently preferred embodiment of the invention uses 8-connected components, but it is con- 
templated that 4-connected components may also be Wentified and used. 

[004S] Because the border 111 of the glyph sticker pattern 110 (Figure 1) is formed from single group of contiguous 
black pixels. It should be observed that the border 111 comprises a single connected component. In a preferred em- 
bodiment of the invention, the white space 1 1 3 around the border 1 1 1 1s sufficiently wide to ensure that the border 1 1 1 
Is not Inadvertently connected with some other portkxi of the hardcopy document to which the glyph sticker pattern 
1 1 0 is applied. Moreover, the border 1 1 lis also sufficiently thick to ensure that It is not Inadvertently broken into discrete 
components by image noise. 

[0046] The connected components within the image are identified by means known In the art (l.e., by starting with 
a single black pixel within the image, recursively kx»tlng all connected pixels until the connected component is fully 
defined, and repeating until all black pixels within the image bekxig to a connected component) . However, other means 
of identifying connected components may also be used to equivalent effect 

[0047] Each connected component is then processed separately. If there are any connected components remaining 
to process (step 611 ), then the method continues as follows. If the connected component is too small (step 612), that 
is, if its width or height is less than a minimum expected value, then the connected component is rejected (step 61 4). 
In an embodiment of the invention, the minimum expected value, used for both height and width, is the smaller of h^i^ 
and w^in to account for possible rotation of the pattern 110. Likewise, the connected component is rejected (step 61 4) 
if it is too large (step 616), and its wkfth or height exceeds the maximum expected value by a substantial amount. In 
an embodiment of the invention, the maximum expected value, for both height and width, is substantially larger than 
the greater of h^^ and w^^ to account for possible rotation. When the pattern 110 (or any rectangle) is oriented at 
an angle, its width and height may appear to be larger than expected, approaching or equaling d^. Accordingly, a 
buffer is built into the latter check. 
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[0048] Eight extreme points, one for each of eight "compass positions," are then selected (step 61 8) from the set of 
pixels comprising the connected component C. Each point has a positton represented by an (x, y) coordinate pair, and 
represents a pixel of the connected component C that extends furthest in the selected direction (north, northeast, east, 
southeast, south, southwest, west, or northwest) within the image plane. Each extreme point is chosen as follows. 

W=(^/v.yN) e Cly„ s y V (x.y) G C 



S= (x^y^ e C I s y V (x.y) G C 



W={x^yje Clx^SxV(x,y)e C 



NE = (x^g y^g) G C I (x^j -y^p) a (x-y) V (x.y) e C 



SE = (Xs£ ys£) G C I (x^e +ysp) 2 (x+y) V (x.y) G C 



S»V= (Xstv, ysw)eCl (xg^ -ys^) s (x-y) V (x.y) G C 



i^NW. y/vw) e C I (x^^y + y^) s (x+y) V (x.y) G C 

[0049] It should be noted that various optimizations can be performed in Identifying the eight extreme points. For 
example, if the connected component C Is broken down Into horizontal runs of contiguous pixels, then only the leftmost 
pixel in each mn need be considered as a candidate for the NW. W, and SkV extreme points, and only the rightmost 
pixel in each run need be considered as a candidate for the WE. E, and SE extreme points. Moreover, if the horizontal 
runs are ordered vertically, then only the endpoints of the uppemiost and lowermost runs need be considered for the 
A/and S extreme points. It should further be noted that, when the border 111 (Figure 1) is oriented precisely at any 45' 
inten/al from upright, then there may be many pixels within the connected component that satisfy one or more of the 
criteria for extremeness set forth above. This Is not a problem; any satisfactory pixel may be chosen. For example, 
there is no single south-most pixel In the border 111 of Figure 1 , so any pixel abng the bottom edge may be chosen. 
Moreover, It is not a problem for a single pixel to be used as multiple extreme points. Referring again to Figure 1 , for 
example, a bottom-right comer pixel 122 may be used as the S, SE, and E extreme points without any impact on the 
described method. 

[OOSO] If the connected component C Is the border 111 (namely, a rectangle with one missing comer), then three of 
the extreme points will contain rectangular comer points (the folded-over comer will be offset), and the other five will 
contain irrelevant information. However, the method described herein has not yet determined whether the connected 
component Cis the border 1 11 , or If so, which extreme points represent its comers; that Is detemilned as set forth below. 
[0051] Consider, then, all eight extreme points. If the distance between any two diametrically opposed extreme points 
satisfies the diagonal length criteria (step 620), then the connected component is a border candidate. That is, if d„i„ 
^|W-£l|<d„ax. or if rfmin ^ ll^-VMI^c'niax. or If d^^ ^NE- SMl|<cf„„ , or If d^i, < USE - NMfl s d„„, then a border candidate 
has been found. Othenwise, the connected component C is rejected (step 614). 

[0052] Based on the position of the diagonal, the other two potential comers of the connected component are then 
identified (step 622) and considered. If the diagonal was found between either the W-Sor £-lfV extremities, then the 
values of N. E, S, and Ware used for p,. ft, ft, and p^. representing the four comer points of the border candidate, 
respectively Similarly, it the diagonal was found between either the NESWot SE-WIV extremities, then the values of 
WE, SE, SW. and AW are used for p,, ft, ft, and p^. the four comer points of the border candidate, respectively 
[0053] Then, relationships anraig the four points p, , ft, ft, and p^ are analyzed to determine whether a rectangular 
shape Is present (step 624). In particular, the distances between p, and ft. ft and ft, ft and p^. and p^ and p, are 
all considered. At least one distance (of the four possible) should approximate the expected width, and at least one 
adjacent (but nof the opposing) distance should approximate the expected height. 
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[0054] That is. if either 



((«'min ^ HPi - PS S Or{W^„ S IIP3 - pj S W^)) 

and 



((/"mln ^ IIP2 - P3II ^ "max) "''Cmin ^ IIP4 " PiU ^ "max)) 

or 



(("mh S IIPi - Pjl S /I^x) O'-C'min ^ IIP3 " Pj ^ "««x)) 

and 

«"'min ^ flP2 - P3II S «'max),<'^ C^min ^ IIP4 " PlU ^ »*'max)) 

20 

Is true, then the connected component C is potentially the border 111 , and It is added to a list (step 626) of possible 
positions. If not, the connected component C is, once again, rejected (step 614). 

[OOSS] As stated above, a list of potential posittons is generated from all border candidates that satisfy the foregoing 
criteria. The entire list is then passed to the decode process (step 514 of Figure 5). Presumably, only one border 
ss candidate will contain valid glyph (or otheO data. If more than one border candidate includes valid data, then the first 
one found is used. 

[0056] It should be noted that the foregoing method of identifying and locating the substantially rectangular border 
111 of the glyph sticker pattern 110, in any orientation and with minor variations in scale, would be applicable to and 
useful in numerous other image-processing applications. It is contemplated that the method illustrated in Figure 6 and 

30 described in detail above may be used to identify and locate nearly any substantially rectangular pattern within a 
digitized Image, either as a part of or separately from the other features and capabilities described herein. 
[0057] Furthemiore. It will be appreciated that minor variations of this method, readily apparent to those skilled in 
the art, may also be used to identify and locate various parallelogram, rhombus, trapezoid, and irregular quadrilateral 
patterns in additksn to rectangles and rectangle-like shapes. In these alternative embodiments, either one or both of 

35 the diagonals can be checked; the two diagonals may have different expected lengths. In addition, the method can be 
extended to identify and locate n-sided polygons, by kJentifying extreme points at 2n evenly-spaced compass positions, 
and thereafter checking for vertices at alternating extreme points. Where n is an odd number, it shoukJ be noted that 
any diagonal found will not be diametric; however any expected distance (or set of expected distances) between ad- 
jacent or non-adjacent vertices, in nearly any n-sided polygon, can be used In a method according to the inventton to 

40 identify the polygon within an image. 

[0058] Figure 7 illustrates the sequence of steps performed in recognizing a glyph field 112 (Figure 1 ) and translating 
it mto binary data. First, the glyph lattce Is detemnined (step 710). Essentially, glyph lattice vectors (Figure 8) along 
with skew and scale information need to be detemnined; this operation is.illustrated in Figure 9 and described in further 
detail below. Second, a seed glyph is found (step 712). The recognitbn process begins by finding a single "good" glyph, 

45 the seed glyph; see Figure 11. Third, all of the glyphs are found (step 714) by recursively examining the seed glyph's 
neighbors (as shown in Figures 1 2-1 3). The lattice vectors direct the search and help perform the mapping from image 
space to a matrix of glyph values. As will be shown beiow, these steps can all be performed regardless of the glyph 
fieW's rotation. 

[0059] Once all of the glyphs have been located, the proper rotation of the glyph field 11 2 is detennined (step 716). 

so As can be seen in Figure 1 , the preferred glyph configuration comprises "slash" patterns at +45° and -45° orientations. 
Accordingly, the glyph field 1 1 2 may appear to be valid at orientatfons of 0°, 90°. 1 80°, and 270°. Therefore, the binary 
pattems represented by the glyph field 11 2 are processed to determine the correct rotation. In one embodiment of the 
invention, a synchronizatbn pattem is embedded into the glyph field 1 1 2; this pattern will only be reconstructed correctly 
when the field 112 is in the correct orientation. For a method of disambiguating the glyph field orientation, see the 

S5 specification and drawings of U.S. Patent No 5,449,895 to Hecht and Stearns, entitled "EXPLICIT SYNCHRONIZA- 
TION FOR SELF-CLOCKING GLYPH CODES," which is hereby incorporated by reference as though set forth in full 
herein. In particular, see columns 15-16 of the '895 patent for a detailed description of rotattonal disambiguation. It 
should be noted that the positkjn of the corner 120 (Figure 1) may also be used in an alternative embodiment of the 
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invention to determine the correct orientation of the glyph field 112. 

[0060] As shown in Figure 8, a glyph image 810 can be defined by two lattice translation vectors 81 2 and 81 4 and 
two (or more) bitmaps 816 and 818 that specify the information that is to be represented at each lattice site. This 
definition is analogous to defining a crystal in solid-state physics. This method of specifying the glyph image allows for 

S a wide variety of structures, but it also provides a scheme for handling arbitrary rotations, scales, lattices, and bitmaps. 
For instance, rotation of the glyph image 81 0 simply corresponds to a rotation of the translation lattice vectors 81 2 and 
814. Just as translation vectors would define the displacement from glyph center to glyph center during construction 
of the glyph image, the translation vectors also define the displacement from glyph center to glyph center during the 
recognition process. Viewing the glyph image as a lattice allows the recognition scheme to handle a wide variety of 

10 glyph formatting schemes while handling rotation and scale variations. 

[0061] The method for determining the glyph lattice is based upon building up an average picture of what the local 
neighborhood of the glyph is. and from this average, determining the lattice vectors. In a preferred embodiment of the 
invention, this is implemented as follows (and as illustrated in Figure 9). 

[0062] In the glyph Image (that is, the area within the border 111), identify some number of pixels in the image as 
IS seed pixels for processing (step 910). These seed pixels may be, for example, spaced on a lattice of W x W pixels 
throughout the Image, where A/ is of the order of 1.0 - 50 (the spacing may be chosen so that there are approximately 
100 seed pixels throughout the glyph image 810). 

[0063] Then, starting at each seed pixel, find the local minimum intensity in the image nearest that seed (step 91 2). 

If no local minimum is found within an appropriate distance of the seed pixel (e.g. 10 pixels), then move to the next 
20 seed pixel. The local minima, i.e. the darkest points in the bitmap, typically correspond to glyphs. 

[0064] Once a local minimum is found, it is determined whether there is sufficient image contrast (step 91 4) between 
■ the region within 1 to 2 pixels of the minimum (i.e., the center), and the region 3 to 4 pixels from the minimum (i.e., the 

surrounding area). If the contrast is too small compared to the average Image Intensity surrounding the local minimum, 

then the niethod abandons further computation with the present local minimum, and moves to the next seed pixel In 
25 the glyph image (if there are any). This step, which may be left out if desired, Is employed to screen out spurtous local 

minima in the image resulting from noise and other non-glyph material. 

[0065] Othenwise, the grayscale Image surrounding the local minimum is added (step 916) to a composite Image, 
which initially Is filled with pixels of value zero. This composite image, which is built up during the processing, thus 
becomes a sum of the bitmap values around a number of local minima In the glyph Image. The composite image is 

30 typically on the order of 20 x 20 pixels In dimension. 

[0066] lfthereareanyseedpixelsremalningtobeprocessed(step9lB),thentheprocessisrepeated. 
[0067] After finding the local minima associated with all of the seed pixels, the composite image is analyzed (step 
920) to determine the average glyph locations. For example, the center of the composite image will necessarily be a 
local minimum, as it is composed of many Images whose centers were minima. The nearest local minima to the center 

35 in the composite image will then correspond to the average nearest neighbor positions of the glyphs in the glyph lattice. 
Determination of these minima in the composite image will therefore result in knowledge of the configuration of the 
glyph lattice; the lattice vectors 812 and 814 can then be derived. 

[0068] The method described above and in conjunction with Figure 9 is very robust, due to the averaging properties 
of building up the composite image. As in any signal averaging procedure, it is expected that in the composite image, 
40 pixels corresponding to the locations of glyphs will sum coherently, while image noise will not. In practice, it has been 
found that the average glyph lattice paranroters of an image may be obtained robustly using the above procedure, 
even when a large number (>50%) of the glyphs are obscured or eradicated by Image noise or ovenmriting. In addition, 
the above procedure works under any amount of Image skew. 

[0069] From the glyphs' lattice vectors 812 and 814. a search direction list 1010 is generated. See Figure 10. The 
<5 search direction list is a list of corresponding relative displacements in image space 1012 and data space 1014 for 
finding neighboring glyphs. From a known glyph in image space, the search direction list 1010 provides informatbn 
on where the neighboring glyphs shoukJ be located. And for each translatton in image space (1016, 1018, 1020, and 
1022), there is a corresponding relative displacement in data space (1024, 1026, 1028, and 1030). Accordingly for a 
known lattice structure, rotation, and scale, the relative movements in image space can be specified without changing 
so the corresponding movement in data space. This method of defining the search allows the process to accommodate 
a wide variety of glyph lattices, rotations, and scales. It shoukJ be noted that the search directton list is also extendable 
to search for second-nearest-neighbor glyphs, or non-rectilinear search pattems. 

[0070] The operations performed in finding a seed glyph for the recognition process will be discussed in connection 
with Figure 11. Finding a good seed glyph is critical. If the seed glyph is actually a spurious center, the process will 
ss quickly fail. Preferably, the process looks for a seed glyph that has at least four neighboring glyphs. This gives some 
assurance that the chosen seed is not spurious. 

[0071] The seed glyph is found by looking at random locations in the images. At a chosen random tocation (step 
1110). a set of con-elation filters is applied over a 3 x 3 neighborhood of pixels adapted to cover the area of a single 
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glyph (step 1 1 1 2). The 3 x 3 neighborhcxxJ accommodates noise and small variations in the glyph lattice. The correlation 
filters that are applied depend upon the image skew determined from finding the lattice vectors 812 and 814 If the 
skew is between -22.5° and 22.5". the following correlation filter kernels are applied: 



1 0 0 
= 010; 
0 0 1 



0 0 1 

0 1 0 

1 0 0 



If the skew is between -45" and -22.5°, the foltowing kernels are applied: 

IS 

"0 0 0] To 1 0" 

^0=1 1 1 ; ^,= 0 1 0 

20 [0 0 oj 0 10 



If the skew is between 45" and 22.5'. the following kernels are applied: 



0 1 0' 
0 1 0 
0 1 0 



"0 0 0" 

1 1 1 

0 0 0 



[0072] The two correlation kernels are applied separately over the 3 x 3 neighbortiood where a glyph is expected. 
9S Since the glyphs typically appear black, the minimum value of the two correlations is observed, and the difference 
between the two correlatkxis is formed (step 1114) and compared to a threshold (step 1116): 



v=min(K;,M3^3)-min(/Ci*l3x3) 

where * denotes correlatton. If. the absolute value of v Is less than a preset ttireshold (step 1 1 1 8) , the location does 
not contain a glyph. If vis positive, the glyph has a value of 0. and if v is negative, the glyph has a value of 1 . 
[0073] Once a glyph Is found, the method looks for neighboring glyphs (step 11 20) using the search direction list 
described above. The method stops looking for a seed glyph once it has found a glyph with four neighboring olyDhs 
(step 1122). . a sa/K 

[0074] The final step in the recognition process is to determine the values of the individual glyphs and place these 
values into a matrix. The invention uses the search direction list (Figure 10) to find new glyphs and to properly map 
their values into a matrix. The process, whfeh is shown in Figure 1 3. begins at the seed glyph. The seed glyph is placed 
into a dynamic data matrix and the location of the glyph in Image space and data space are placed onto a search FIFO 
(first-in-first-out) list (step 1310). 

[0075] The search FIFO list controls the clustering. As new glyphs are found, they are placed at the end of the search 
FIFO list. Essentially, the search FIFO list contains the kxations in the image that need to be searched for neighboring 
glyphs. When the search FIFO list is empty, the process stops. 

[0076] Beginning with the seed glyph, if any glyphs are present in the FIFO list (step 1 31 2). the method pulls a glyph 
locatran from the FIFO list (step 1314). Its value Is determined (step 1316) and placed into the data matrix. The value 
of the glyph is determined by using the same correlatton kemels used to find the seed glyph. The method then searches 
outward (step 1318) looking for neighboring glyphs using the search direction list (see Figure 1 2(a)); neighboring glyphs 
are placed into the FIFp list (step 1320). The method generates a cluster of known glyphs that grows outward as it 
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searches for new glyphs around the perimeter of the cluster (Figure 12(b)). The outcome of the procedure is a data 
matrix full of data values 1210 (Figure 12(c)) representative of the glyph Image 810. 

[0077] This method has several desirable features. First, it can handle groups of glyphs with arbitrary boundaries. 
The glyph block need not be rectangular. Second, it provides a clean method of dealing with scale and rotation varia- 
tions. Finally, it is tolerant to irttensity gradients and markings placed on top of the glyphs. It should be noted that 
although a FIFO list is used in the presently preferred embodiment of the invention, other search methods, including 
those that use a LIFO (last-in-first-out) stack or a list based on some other ordering scheme, such as position within 
the image, can also be used with similar effect. 

[0078] Finally, as discussed above, the data matrix is used as shown in Figure 14. First, the data matrix is rotattonally 
disambiguated as described above (step 1410). The binary values in the data nnatrix are re-mapped accordingly (step 
1412), which may involve re-ordering the matrix and inverting the data values, as necessary based on the lotatfon. 
The sen/ice code is extracted from the resulting data (step 1414). as is the user code (step 1416). and the inventkxi 
handles the sen/ice as discussed above in conjunction with Figure 3. 



Claims 

1 . A method for processing a document based on information in a user interface tag, comprising the steps of: 

scanning the document to produce an image representative of the document; 
Ipcating the user interface tag in the image; 
decoding data represented in the user iriterface tag; 
associating the data with a sen/ice; and 
perfomning the specified sen/ice. 

2. A method according to claim 1 , further comprising associating the data with a user Wontity. 

3. The method of claim 1 or claim 2, wherein the step of locating the user interface tag comprises the steps of: 

identifying a connected component in the image; 

finding a plurality of extreme points within the connected component; 

determining whether a diagonal length may be present between two of the plurality of extreme points; 
if so, identifying corners of a border candidate; and 

deterriiining whether a correctly-dimenskxied rectangular shape Is defined by the comers. 

4. The method of any of the preceding claims, wherein the step of decoding the data comprises the steps of: 

detemnining a lattice of glyphs represented in the user interface tag; 
identifying a seed glyph within the lattice; 
finding all glyphs within the lattice; 
identifying the rotatkm of the lattice; and 
converting the glyphs to binary data. 

6. A method for creating a user interface tag for use with a tag-based document service system, comprising the steps 

of: 

receiving user information representative of a user's identity; 
creating an identity code based on the user infomnation; 
storing the user infomiatkxi and the identity code in a database; 
generating a printed data code including the identity code; and 
printing a user interface sticker bearing the printed data code. 

6. A user interface tag bearing a machine-readable printed data code, wherein the tag is adapted to be associated 
with to a hardcopy document for scanning by a document processing system, and wherein the data code comprises 
an Identity code representative of a user's kJentity. 



7. A method for locating a substantially rectangular pattern in a digitized image, the pattem having an expected 
diagonal length, comprising the steps of: 
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receiving data representative of the digitized image; 

identifying a connected component in the image; 

finding a plurality of extreme points within the connected component; 

determining whether the expected diagonal length Is present between two of the plurality of extreme points; 
if so, locating comers of a pattern candidate; and 

determining whether a correctly-dimensioned substantially rectangular shape is defined by the corners. 

8. The method of claim 7. further comprising the steps, after identifying a connected component, of: 

determining whether the connected component is substantially narrower or shorter than the srnaller of the 
expected width and the expected height, and if so, rejecting the connected component; and 
determining whether the connected component Is substantially taller or wider than the greater of the expected 
width and the expected height, and if so, rejecting the connected component. 

9. The method of claim 7 or claim 8, wherein the finding step comprises identifying eight extreme points in positions 
corresponding to compass positions representing north, northeast, southeast, south, southwest, west, and north- 
west, the method further comprising the step of selecting a first corner point, a second comer point, a third comer 
point, and a fourth comer point from the eight extreme points such that the two points defining the expected diagonal 
length but no two adjacent compass positions are included in the comer points. 

10. The method of claim 9, wherein the step of detemilning whether a correctly-dimensioned substantially rectangular 
shape Is defined comprises the steps of: 

determining whether either a distance from the first comer point to the second comer point or a distance from 
the third comer point to the fourth comer point is substantially equal to the expected width, and whether either 
a distance from the second comer point to the third comer point or a distance from the fourth corner point to 
the first comer point is substantially equal to the expected height, and if both width and height criteria are 
satisfied, indicating that a correctly-dimensioned substantially rectangular shape is defined by the four corner 
points; and 

determining whether either a distance from the first corner point to the second comer point or a distance from 
the third comer point to the fourth corner point is substantially equal to the expected height, and whether either 
a distance from the second corner point to the third corner point or a distance from the fourth corner point to 
the first comer point is substantially equal to the expected width, and if both height and width criteria are 
satisfied, indicating that a correctly-dimensioned substantially rectangular shape is defined by the four comer 
points. 
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